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UNIVERSAL QUALITY MEASUREMENT SYSTEM 
FOR MULTIMEDIA AND OTHER SIGNALS 

BACKGROUND OF THE INVENTION 

This invention relates to a system for measuring a signal quality, and more 
5 particularlytoasystemformeasuringqualityofmultimediasignalscommunicated 

via a network. 

Numerous processing and transmission methods have been devised for the 
communication of voice, audio, video, and multimedia information. These types 
of signals are collectively referred to herein as "voice, audio and video" (VAV) 
,0 stgnals. VAV stgnals may be processed in many different ways before bemg 
emitted to a receiver. For example, a VAV signal may be dig.trzed, 
compressed, and modu.ated onto a carrier. The receiver then must transform the 
received signal back into a perceptible representation ofthe VAV stgnal. 

Traditionally, received VAV signal quality is evaluated by subjects 
15 testing However, this type of test is no, practical as an in-service testing method, 
nor does this type of test produce consistent, reproduce results. Signal quahty 
evaluation is further complicated when VAV communication is incorporated mto 
very complex transport systems such as cellular code division multiple access 
(CDMA) systems or wideband CDMA systems. Different, but equally difficult 
20 challenges are presented when evaluating VAV quality in such non-determmtsttc 
environments as the Internet. 

It is known to evaluate signal quality indirectly by measuring parameters 
such as signal-to-noise ratio (S/N), carrier-to-interference ratio (C/I), lost packet 
rate and bit error rate (BER). However, i, is difficult to relate these parameters to 
25 user perception ofthe quality of received VAV signals, particular.y when such 
signals are highly compressed or processed. Also, these parameters are no. well- 
suited to reflect VAV quality correctly and accurately over time. 

Algorithms to estimate perceptual speech quality are known. For example, 
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International Telecommunications Union standard P.861 is an objective algorithm 
that can be used to automatically compute a qualitative figure such as a Mean 
Opinion Score (MOS) for speech transmission. However, these types of tests 
require both an original and a test signal to be statically analyzed by an item of test 
5 equipment. Thus, if quality evaluation is to be done in the field with real signals, 
the test equipment must incorporate network architecture and protocols into its 
hardware, software, or firmware. Often, the network architecture is not available, 
or it may be difficult or expensive to implement. Proprietary restrictions may even 
preclude its implementation. 
10 In view of the above, methods and apparatus for evaluating quality of 

received VAV signals that do not require explicit knowledge of the operation and 
configuration of a network, such as the transport technology used and the network 
access technology, would be desirable. It also would be desirable if the methods 
and apparatus provide real-time or non-real-time automated measurements of 
15 quality, so that interactive field testing, for example, could be performed. In 
addition, a common test platform for various types of VAV signals that easily 
connectstoan equipment under test would be advantageous. Furthermore, methods 
and apparatusteprovide reliable, 

quality of service measurements for one-way, multi-hop, or round-trip 
20 measurements would be desirable. 

BRIEF DESCRIPTION OF THE INVENTION 

In one embodiment, the invention is a method for measuring perceptual 
quality of voice signals, audio signals, audio-video signals, and multimedia signals 
in communication equipment. A processed test signal is communicated via a 
25 network from a first device to a second device. The processed test signal is then 
received by an equipment under test, which further processes the processed test 
signal to recover a representation of the test signal. The recovered test signal is 
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then objectively analyzed to determine a measure of perceptua. qual.ty by 
comparingtherecovered.es, signal to a pre-stored representation of the .est s.gnal. 

, n another embodiment, the invention is a quality measurement urn, that 
attaches to equipment under test to identify and eva,ua,e quality of recovered test 
signals The test signals are .hose communicated from a remote device and those 
communicated by the equipment under test. In the latter case, the quahty 
_men.unitis conjured to communicate wi.h the equipment under tes, and 
the network, so .hatthequalitymeasurementunititselfreceivesarepresentauonof 

the communicated test signal by the equipment under test. 

it will be seen that, in methods and apparatus of this invention, quahty of 
received VAV signals are evaluated without an explicit knowledge of the network 
via which .he equipmen. under .es. communica.es. Moreover, bom real-.ime and 
non-real-time quality analysis is possible, utilizing a common tes, platform for 
various types of VAV signal, The invention is applicable .oavarieiy of d.fferen. 
,5 <y P es of quantitative quality of service measurements, inc.uding one-way, mum- 
hop, and round-trip measurements. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of an embodiment of a quality measurement 
system for VAV signals suitable for one-way quality measurements in accordance 
20 with the invention. 

Figure 2 is a flow char, of an embodiment of a quality measurement method 

applicable to the system of Figure 1 . 

Figure 3 is a block diagram of an embodimen. of a quality measurement 
sys.em sutable for round-trip quality measurements in accordance with the 
25 invention. 
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Figure 4 is a block diagram of an embodiment of a quality measurement 
system suitable for two-way quality measurements in accordance with" the 
invention. 

DETAILED DESCRIPTION OF THE INVENTION 

5 In one embodiment and referring to Figure 1 , a quality measurement system 

10 for VAV signals is shown in conjunction with equipment under test 16 and a 
network 22. Quality measurement system 1 0 includes a local quality measurement 
unit (LQMU) 12. A port 14 of LQMU 12 is coupled to an equipment under test 
(EUT) device 16. In one exemplary embodiment, EUT 16 is a mobile or cellular 

10 telephone, and port 14 is electrically coupled to EUT 16 using, for example, a set 
of "hands-free" or "car kit" electrical terminals (not shown). The invention is not 
limited in applicability to mobile and cellular telephones, however. 

EUT 16 transmits and receives, via communication paths 18 and 20, 
respectively, information to and from a network 22. For example, in the case of a 

15 cellulartele P hone,network22comprisesawirelessnetwork24andcommunication 

paths 1 8 and 20 are radio links. A remote reference data storage unit (RRDSU) 26 
is also operatively coupled to network 22. In one exemplary embodiment, RRDSU 
26 is a device for recording and playing reference signals that is coupled to wireless 
network 24 via a wire-line network 28, for example, a telephone answering 
20 machine. A network interface adapter (NIA) 30 is provided to condition signals 
output from RRDSU 26 into a form appropriate for transmission to network 28. 
Similarly, signals transmitted from network 28 to RRDSU 26 are converted by NIA 
30 into a form suitable for processing by RRDSU 26. 

In one embodiment, a local storage in the form of a local reference data 
25 storage unit (LRDSU) 32 is provided for LQMU 12 and is coupled to LQMU 12 
via a second port 34. In another embodiment, local storage 32 may be internal to 
LQMU 12. In other embodiments, local storage, such as electronic or magnetic 
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m emory is interna. ,o LQMU 12. LQMU 12 provides an indication of a quahty 
m easure 36 of a test signal received from RRDSU 26 via network 22 by EUT 16. 

Knowledge of protocols, signaling, and configuration of network 22 
required for the practice of the invention are isolated in EUT ,6, a pre-existing 
piece of equipn**. and inNIA 30. N1A 30, in many instances, may be a read.ly 
a vai,ab 1 edevicesupp 1 iedbyano P eratorofnetwork28orane.ec,ronicsdis«ributor. 

The present mvention aUows different types of network 22 to be accommodated 
without requiring specific information concerning the nature of network 22 to be 
known or available to RRDSU 26, LQMU 12, or local storage 32. This allows 
RRDSU 26 to be coupled via N1A 30 to a publicly- or user-accessible servce 

access point of network 22. 

in one embodiment, port 14 and LQMU 12 are configured to collect 
received test signais from EUT 16 so that LQMU 12 is responsive to signals 
recetved by EUT 1 6. In one embodiment, port 14 and LQMU 12 are configured to 
con.ro. EUT 16 to transmit up.ink signals via radio link .8. LQMU 12 is also 
configured to analyze thereceived.es, signals to perform an objective measurement 
of the quahty of these signals. In one embodiment, embedded algorithms withm 
LQMU 12 are configurable and se.ectab.e for one or more types of VAV s.gnals. 

RRDSU 26 is, for example, a storage unit including a recorder and a player 
(no, shown separate.y in Figure 1). The recorder is, for example, a 
readab,e/wri,eab.e mass storage device such as a hard disk drive, or any other 
device suitable for recording VAV signals. The player, in one embodiment, 
accesses the recorded VAV signals, which include reference VAV signals, and 
p,ays them back. RRDSU 26, in one embodiment, includes a controller module 
(not shown) that is controllable via signais received via network 22 ,o command 
RRDSU 26 ,o perform such tasks as start, stop, play, record, pause, seek, 
timestamp, etc. In one embodiment, RRDSU 26 comprises a digital answenng 
machine* perform these function, In one embodiment, RRDSU 26 is configured 
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to transmit a test signal that is selected from voice, audio, video, and multimedia 
test signals. 

In one embodiment and referring to Figure 2, quality measurement system 
10 is utilized for testing EUT 16 in conjunction with network 22 comprising a 
wireless telephone network 24 to evaluate one-way downlink end-to-end VAV 
quality, i.e., the quality of a signal received via radio link 20 by EUT 16 from a 
base station (not shown) of network 24. Initially, LQMU 12 is turned on 100 and 
connected to EUT 1 6 so that it is responsive to receipt of a test message by EUT 16. 
EUT 16 places a call 102 to RRDSU 26 by dialing RRDSU telephone number so 
that RRDSU 26 transmits a test signal to EUT 16. In another embodiment, when 
EUT 16 is configured to communicate via a network other than a telephone 
network, an appropriate signaling technique is substituted for placing a telephone 
call to RRDSU 26. In other embodiments such signaling may also be performed 
manually, for example, by dialing a number on a keypad (not shown) of EUT 16. 

in one embodiment, RRDSU 26 is configured as a voice mailbox, so that 
dialing 102 RRDSU also includes dialing a voice mailbox of RRDSU 26 and 
retrieving 104 a test signal as a voice mail message. Prior to transmission of the 
test signal, for example, a voice message, the test signal is processed by NIA 30 so 
that it is in a form suitable for transmission over network 28. LQMU 12 then 
l attempts to synchronize itself 106 with the test signal. Synchronization search 1 06 
is continued until synchronization 108 is achieved. In one embodiment, when 
synchronization 108 is not achieved within a determined period of time 110, an 
error condition 1 12 is signaled. In one embodiment, by synchronizing 108 with the 
test signal, LQMU 12 analyzes signals received by EUT 16 to determine that a test 
5 signal has been received. It will be understood that EUT 16 performs further 
processing of signals received via network 24 to recover a representation of the test 
signal that is analyzed by EUT 16. This further processing, among other things, 
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makes it possible for EUT 16 to be independent of the signal encodings and 
network protocols utilized by network 22. 

After acquiring synchronization 108, the recovered test signal is rece.ved 
„4 by LQMU 12. The recovered test signal is compared to a pre-stored 
re prese„,at,on of a reference signal in LRDSU 32 and a quality of service (QoS) 
_ is determined U6 to provide an objective measure of received s.gnal 
qualit y This measure, or a representation of it, is displayed .18, for example, on 
a display screen (not shown). In another embodiment, the measure of quahty ts 
rec orded for future display or analysis. In one embodiment, a predetermmed 
nomberoftests are performed 120 during a call to RRDSU 26. The call from EUT 
,6.o RRDSU26can be termmated 122 either manually or automatically when the 
determined number of test or tests are complete. 

,„ on e embodiment, if LQMU 12 does not have sufficient processmg power 
t0 eva.uate quality of a received test signal as it is being received by EUT .6, the 
receivedtest signal is stored in LRDSU 32 for later, off-line quality evaluation, for 
example, after the call from EUT 16 to RRDSU 26 is terminated. 

,„ ye, another embodiment and referring to Figure 3, an up-link tesung 
system 200 does not require an RRDSU 26. Testing system 200 is shown in Figure 
3 in conjunction with network 22 and EUT 16. In this embodiment, EUT 16 .s a 
cellular telephone transmitting via radio link 1 8 to a wireless network 24. LQMU 
,2 receives signals from NIA 30 via wire-line network 26. In one embod.ment, 
MA 30 is a modem, such as a computer modem, with which LQMU 12 .s 
configured for communication. EUT 16 dials the telephone number of NIA 30 to 
initiateatest.LQMU12the»injectsalocal-to-remote test signal into EUT16,for 

example, via a hands-free kit, which is returned via network 22 and NIA 30 to 
LQMU 12. LQMUnthenperformsaqualityofservice evaluation on therecetved 

signal. 
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.nanother embodiment and referring. o Figure 4, a. wo-way testing system 
300 is prov.ded having a master quality measurement unit (MQMU, 302 and a 
s ,ave quaiity measurement unit (SQMU) 304. Two-way testing system 300 ,s 
sh ow„ in a configuration in which testing of EUTs 306 and 308 is performed m 
conjunction with a network 3,0. MQMU 302 is configured to mutate and 
c „„rd 1 na.ecom m unicationbetweenEUT306andEUT308,forexample,«erm,nas 

providing VAV capacities, via a network 3.0, for examp.e, the .ntemet, and ,0 

Otherwise, MQMU 302 and SQMU 304 are configured simi.arty to LQMU,2.n 
that MQMU 302 is coupled to BUT 306 and reference data storage un,t (RDSU 
3,2 a„dSQMU304isc„up,edtoEUT308and RD SU3.4. Both MQMU 302 and 
SQMU 304 are configured to provide ouaiity measures of received stgna , 
MQMU 302 provides a quality measure 3 1 6 of a signal transmitted from EUT 308 
via network 3 .0 to EUT 306, while SQMU 304 provides a quality measure 3, 8 of 
signals transmitted from EUT 306 to EUT 308 via network 310. In one 
eldimen, timestamping of qua.ity measures is determined so tha, qual.ty 
measures 3,6 and 3,8 can be corrected and post-processed by a post-processtng 

module 320 at a later time. 

,„ one embodiment, neither LQMU ,2, RRDSU 26, MQMU 302 nor 
SQMU 304 require adirec, connection with any network. Forthisre.cn, eachof 
these devices can operate in a manner that is independent of the network typ. 
i„c,udingi,scha ra cte ri sticsandcor^gura,ion,a,,owingitsuseinc 0 njunc, 1 onw 1U1 

^proprietary and non-proprietary networks. Asaresult, the present tnventton 
may be practiced in conjunction with any of a variety of different types of 
networks, including s W i,ched-circui,, packet-switched, ft ame-re,ay, Interne, 
protocol, andasynchronoustransfermode(ATM)networks.Thepresen.,nven n o„ 

isa,so suitable for a variety of network, ransporttechno.ogies, inducing wneless, 
wired and sateUite networks. Furthermore, components of the present tnventton 
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may be designed to connect to existing terminals or ports of communion 
equipment and to publicly or user available service access points of networks. 
Thus no special set-ups or modiftcationsofthe network orofthe equipment under 

test need be required. In various embodiments, some or all of LQMU 12, MQMU 
5 302 and SQMU 304 are configured to be non-intrusive to permit in-service 
deployment. 

Inoneembodiment,atleastoneofLQMU12,MQMU302andSQMU304 
are implemented utilizing digital signal processing to achieve real-time quality 
evaluation. The quality measurements obtained are used to adaptively adjust 
10 network parameters, if network architecture or protocol information is available. 

In other embodiments, voice quality assessment is improved by selecting, 
for example, adaptively or automatically, an appropriate algorithm for quality 
assessment. For example, multimedia signals transmitted via CDMA and Voice 
Over IP are better evaluated by algorithms that are different from standard quality 
15 measurement algorithms such as International Telecommunications Union (ITU) 
standard P.861 Perceptual Speech Quality Measurement (PSQM) or Measuring 
Normalizing Blocks (MNB), another ITU-developed measuring algorithm to 
measurevoicequality off-line using files stored in a computer. In one embodiment, 
at least one of LQMU 12, MQMU 302 and SQMU 304 are programmable for 
20 different system applications. In one embodiment, a variety of test signals are 
selected, either manually or under control of at least one of LQMU 12, MQMU 302 
and SQMU 304. For example, artificial voice-like signals designed by standard- 
generating bodies such as ITU may be selected. 

In one embodiment, storage 32, 310, and 312 include any of several 
25 different types of storage media or a relay device such as one that communicates 
to another device having storage media. 

In one embodiment, at least one of LQMU 12, MQMU 302 and SQMU 304 
are operatively coupled to at least one of a global positioning satellite (GPS) 
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receiver (not shown) and a digital map (not shown) to automatically generate a 
quality of service map for example, test signals for a wireless telephone network are 
transmitted repeated to a moving EUT 16, 306, or 308, and performance 
measurements are performed and analyzed as a function of position. 

It will be evident to those skilled in the art that many other modifications 
are possible within the spirit of the invention. Therefore, the scope of the invention 
should be determined by reference to the claims appended below and their 
equivalents. 
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CLAIMS: 

1 . A method for measuring perceptual quality of at least one of voice 
signals, audio signals, audio-video signals and multimedia signals in 
communication equ.pment in an operational environment comprising the steps of: 
5 transmitting a processed test signal via a communication network from a 

first device, wherein the test signal is at least one of a voice signal, audio signal, 
audio-video signal, and multimedia signal; 

receiving the processed test signal utilizing a second device; 
processing the processed test signal to recover a representation of the test 
10 signal; and 

analyzing perceptual quality of the recovered test signal by comparing the 
recovered test signal to a pre-stored representation of the test signal. 

2. A method in accordance with Claim 1 wherein the first device 
comprises a remote data storage unit (RRDSU) operatively coupled to the network 
15 to communicate a processed test signal therethrough, and the second device 
comprises an item of equipment under test (EUT) operatively coupled to the 
network for receiving the processed test signal and to a quality measurement unit 
(QMU) for analyzing quality of the recovered test signal; 

said method further comprising the step of analyzing signals recovered by 
20 the EUT to determine that a test signal has been recovered prior to said step of 
analyzing quality of the recovered test signal. 

3. A method in accordance with Claim 2 further comprising the step 
of the QMU displaying an indication of the quality of the recovered test signal. 



BNSDOCID <WO _0O724S3AU_» 



WO 00/72453 



PCT/US99/11529 



20 



-12- 

4. A method in accordance with Claim 2 further comprising the step 
of contacting the RRDSU from the EUT via the network prior to said step of 
communicating the processed test signal. 

5. A method in accordance with Claim 4 wherein the network 
comprises a telephone network, the EUT comprises a telephone, and said step of 
contacting the RRDSU from the EUT via the network comprises the step of dialing 
a phone number of the RRDSU. 

6. A method in accordance with Claim 5 wherein the step of contacting 
the RRDSU from the EUT further comprises the step of dialing a voice mailbox 
access code. 

7 . A method in accordance with Claim 2 wherein said step of analyzing 
quality of the recovered test signal is performed while the processed test signal is 
being received by the EUT. 

8. A method in accordance with Claim 2 further comprising the step 
of storing the recovered test signal, and wherein said step of analyzing quality of 
the recovered test signal is performed after the test signal has been stored. 

9. A method in accordance with Claim 8 wherein said further 
comprising the step of disconnecting the EUT from the network prior to said step 
of analyzing quality of the recovered test signal. 

10. A method in accordance with Claim 2 wherein said step of analyzing 
quality of the recovered test signal is performed digitally. 
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! 1 A method in accordance with Claim 2 further comprising a step of 
selecting a quality assessment method in accordance with a characteristic ofthe 
network. 

12 A method in accordance with Claim 2 wherein the network is a 
wireless telephone network, said steps of communicating a processed test signal 
andofanalyzingquahty ofthe recovered testsignal are performed repeatedly, and 
said method further comprises the steps of: 

moving the EUT so that the repeated communications ofthe processed test 
signal are received in different positions ofthe EUT; and 

analyzingquality ofthe recovered test signals as a function of the different 

positions of the EUT. 

13 A method in accordance with Claim 2 wherein processed test signal 
communicatedbythe first deviceto the second device is a processed representation 
of a first test signal, and further comprising the steps of: 
; injecting a representation of a second test signal into the EUT utilizing the 

QMU; 

further processing the second test signal utilizing the EUT; 
communicating the processed second test signal from the EUT to the first 

device via the network; and 

tether processing the communicated, processed second test signal to 
recover a representation ofthe second test signal utilizing the firs, device; and 

objectively analyzing perceptual quality of therecovered second test s.gnal 
by comparing the recovered second test signal to a pre-stored representation of*, 
second test signal. 
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1 4. A method in accordance with claim 1 3 further comprising separately 
storing results of the analysis of the quality of the recovered first test signal and of 
the recovered second test signal, and associating time stamps with each of the 
separately stored results for later correlation and further processing of the separately 

5 stored results. 

15. A method in accordance with Claim 1 wherein the first device 
comprises equipment under test (EUT) and the second device comprises a quality 
measurement unit (QMU) operati vely coupled to the EUT for injecting a signal into 
EUT for transmission as the test signal and to the network for receiving the test 

10 signal; further comprising the steps of: 

injecting a representation of the test signal in to the EUT utilizing the 

QMU; 

further processing the test signal utilizing the EUT; and 
transmitting the processed test signal to the QMU via the network; and 
15 wherein the step of analyzing quality of the recovered test signal comprises the step 
of further processing the received processed test signal to obtain a recovered 
representation of the test signal, and the step of analyzing quality of the recovered 
test signal by comparing it to a pre-stored representation of the test signal. 
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lb . A quality measurement device for measuring quality of signals 
transmitted via a communication network to equipment under test (EUT), said 
quality measurement device configured to: 
communicate with the EUT; 
5 analyze signals received and recovered by the EUT to identify recovered 

test signals; and 

objectively analyze perceptual quality of said identified and recovered test 

signal. 

17. A quality measurement device in accordance with Claim 16 further 
10 configured to analyze at least one type of said test signal selected from the group 

consisting of voice signals, audio signals, video signals, and multimedia signals. 

18. A quality measurement device in accordance with Claim 1 7 further 
configured to display an indication of the quality of said identified and recovered 
test signal. 



15 



19. A quality measurement device in accordance with Claim 17 further 
configured to synchronize with test signals recovered by the EUT to identify said 
recovered test signal. 

20. A quality measurement device in accordance with Claim 1 7 further 
configured to analyze quality of said identified and recovered test signal while a 

20 processed said test signal is being received by the EUT. 

21. A quality measurement device in accordance with Claim 1 7 further 
comprising local storage, said quality measurement device further configured to 
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store said identified and recovered test signal in said local storage and to analyze 
said identified and recovered test signal after storage. 

22. A quality measurement device in accordance with Claim 2 1 further 
configured to effect disconnection of the BUT from the network prior to said 

5 quality measurement device analyzing said identified and recovered test signal. 

23. A quality measurement device in accordance with Claim 1 7 further 
configured to select a quality assessment method for objectively analyzing 
perceptual quality of said identified and recovered test signal in accordance with a 
characteristic of the network. 

10 24. A quality measurement device in accordance with Claim 20 further 

configured to communicate with a wireless telephone as the EUT. 

25. A quality measurement device in accordance with Claim 20 further 
comprising a local reference data storage unit (LRDSU) having a representation of 
said test signal stored therein, and further configured to control the EUT to 

15 communicate and a processed representation of said test signal via the network. 

26. A quality measurement device in accordance with Claim 25 further 
configured to record indications of objective perceptual quality of said recovered 
and identified test signal and to timestamp said recorded indications of quality. 

27. A quality measurement device in accordance with Claim 25 further 
20 configured to receive, via a network, a representation of said test signal 

communicated by the EUT under control of said quality measurement device, and 
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to objectively analyze perceptual quality of said test signal communicated via the 
network to said quality measurement device. 

28. A system for measuring perceptual quality of at least one of voice 
signals, audio signals, audio-video signals and multimedia signals in 
5 communication equipment in an operational environment comprising: 

a transmitting device configured to transmit a processed test signal via a 
communication network, wherein the test signal is at least one of a voice signal, 
audio signal, audio-video signal, and multimedia signal; 

a receiver configured to receive the processed test signal via the network 
10 andtoprocesstheprocessedtestsignaltorecoverarepresentationofthetestsignal; 

and 

a quality measurement unit (QMU) operatively coupled to the receiver and 
configured to analyze perceptual quality of the recovered test signal by comparing 
the recovered test signal to a pre-stored representation of the test signal. 

15 29. The system of Claim 28 wherein the transmitting device further 

comprises a remote data storage unit (RRDSU) operatively coupled to the network 
to communicate a processed test signal therethrough, and the QMU is configured 
to determine that a test signal has been received prior to analyzing quality of the 
recovered test signal. 

20 30. The system of Claim 29 wherein the QMU comprises a display 

configured to display an indication of the quality of the received test signal. 
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